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ABSTRACT 



This research, performed during 1998 and 1999, had as its 
purpose the development and validation of a course/instructor evaluation 
instrument specifically for use with graduate and undergraduate courses 
delivered online via the World Wide Web. The primary goal of the study was to 
develop an instrument that would collect valid perception data from students 
at West Texas A&M University concerning technology of delivery, course 
content, and instructor- related aspects of courseware delivered via the 
Internet. To this end, an instrument was developed using the input of two 
panels of university faculty, administrators, and staff and repeated pilot 
studies to collect data on the validity and reliability of the instrument. 
Feedback received from the expert panels, short- and long-term pilot studies, 
and analysis of the collected data provided preliminary evidence to support 
the validity and reliability of this Internet delivered course-specific 
evaluation instrument. (Contains 13 references.) (Author/MES) 
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Abstract: Research was performed during the 1998 and 1999 calendar years that 
had as its purpose the development and validation of a course/instructor 
evaluation instrument specifically for use with graduate and undergraduate 
courses delivered online via the World Wide Web. The prinaary goal of the 
study was to develop an instrument that would coUect valid perception data fix>m 
students at West Texas A&M University concerning technology of delivery, 
course content, and instructor related aspects of courseware delivered via the 
Internet To this end an instrument was developed using the input of two panels 
of university faculty, administrators, and staff, and repeated pilot studies to 
collect data on the vahdity and reliability of the instniraent. Feedback received 
fix>m the expert panels, short and long'-term pilot studies, and analysis of the 
collected data provide preliminary evidence to sipport the v^dity and 
reliability of this Internet delivered course-specific evaluation instrument 



Introduction 

As the increased use of instructional technology in teaching becomes ever more necessary in higher 
education, the use of computer technologies to deliver course content to students constitutes the new 
fi'ontier. Because the possession of a university degree in the latter part of the twentieth century h as become 
more and more essential to career success, access to a university e^cation for all that desire it has become 
a priority for umversities large and small. One method of providing such access to students who have not 
previously had the opportumty to attend a university is to deliver course work to the student through the use 
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of newly available Internet based technologies. The delivery of courses in post-secondary education via 
technological means, while not new, has gained greater impetus through the expanded use of Internet 
technologies that center on the World Wide Web. Through the use of the World Wide Web, undergraduate 
and graduate courses can be delivered without any limits as to the geographical l^ation of the student. 
Johnstone (1997) has found that students who elect to take courses via technological means have an 
expectation of convenience and ease of access. This desire on the part of trachtional and non-traditio^ 
students for higher education delivered to them on their own scheme has driven the strong growth in both 
institutions of higher education offering such electronically delivered courseware, and the range of courses 
offered. Twigg (1994) finds that non-traditional students in particular define their role as a participant in 
hi^er education differently than their younger counterparts. Older students tend to view themselves as 
discriminating consumers of an educational product. Because of this view these students do not view 
themselves as limited to what professors and administrators of higher education think they need 

In addition to the differing perceptions and expectations of students for on-line delivered courseware, the 
medium of delivery also plays a role in shaping how students’ perceive their experience as a student in this 
new type of learning experience. While several authors (Tait 1993; Dringus 1995; Caswell & Hobson 
1996; Johnstone 1997; and Magalhaes & Schiel 1997) have addressed the need for high quabty evaluation 
of courses delivered at a distance, there is currently a dearth of research studies publi^ed that offer 
validated evaluation survey instruments specifically written for World Wide Web delivered courseware. It 
is the above listed differences in perceptions by pc^tial consumers of distance delivCTed coursew^ that 
makes valid evaluation of student perceptions and experiences in these courses essential to university 
administration and faculty. 

This study took place during 1998-1999 academic years at West Texas A&M University in Canyon, Texas. 
West Texas A&M University is a member of Texas A&M University System that serves the p anh a n dle 
region of Texas. This region of the state of Texas has an ^jproximate population of 700,000, with the 
university having an ^xproximate enrollment of 6,800 students. In the fall semester of 1997, West Texas 
A&M University implemented four courses for online delivery through the World Wide Web with a total 
enrollment of ^proximately 125 students. Currently (fell semester 1999) 33 different courses are offe^ 
with a total enrollment of approximately 660 students in initial enrollment West Texas A&M University 
cunently plans to make available 25% of its course offerings via World Wide Web delivery by the year 
2003. 

Of the man y challenges faced by universities and instructors who develop and offer online coursware, 
course evaluation may be among the most difficult challenges to address. In this ^y, a new instrument 
was developed to collect course evaluation data for courses that axe delivered online through the World 
Wide Web. The first phase of the development process involved the associate Dean of Information 
Technology, research specialists, and faculty who have desired and facilitated online courseware. The 
new instrument developed as a result of this process was designed to collect data concenmg student 
perceptions of course content, instruction, and delivery. The second phase of fee project involved analysis 
of fee data gathered from participants in fee study to determine internal reliability and fee benefits and 
limitations of using fee Internet to conduct online course evaluations. 

During fee first three semesters feat courses were offered online at West Texas A&M University, no course 
evaluation instruments were distributed for fee collection of student perceptic^ as to course and instructor. 
The university had used (and still uses) a Course/Instructor Evaluation Questionnaire (CEIQ) instrument 
which was an adaptation of fee Aleamoni Couise/Instructor Evaluation Questionnaire (Aleamoni 1978). 
This instrument had been used to gather student perceptions as to course and instructor for some ten years 

prior to the offering of online courses at WTAMU, but had not yet been appHed to online courses due to fee 

early stage of development and inq)lementation of this new instructional mediunL 

Much research has been written on fee validity and reliability of using standard survey questionnaires to 
assess student perceptions of instruction and course content quality (Costin, Gieenough, & Menges 1971 ; 
Centra 1973; Cashin 1988, 1990, 1995; Abrami, D’ApoUonia, & Cohen 1990). However there is very much 
less literature feat deals specifically wife either fee development or validation of online course-specific 
student evaluation instruments. The research team began their investigation wife an Internet search for 




BEST COPY AVAILABLE 



190 



instruments currently lased to evaluate online courseware. This search conducted in January 1 999 found less 
than 5 such instruments tlat were accessible. The majority of the instruments located and evaluated were 
structured using the familiar Likert type questions that sought to quantify student perceptions of perceived 
course and instructor quality. A similar search conducted in late November 1 999, found a much larger 
number of such evaluative instruments available for study, although their quality ranged fiom 
comprehensive to perfunctory. Due to the lack of existing instrumentation available for adaptation at the 
beginning of the study, it was determined that the development of a new instrument would be necessary. 

This newly developed instrument must be valid and reliable, and the data collected through the use of this 
mstrument should be able provide data that will be valuable for formative and summative evaluation of 
coursew^. It is essential that instructors whose online teaching effectiveness is being evaluated through 
such an mstrument have confidence in the feet that the instrument being used to evaluate their teaching can 
actu^ly gather data that is validly measuring their level of success with students in this new tpar-hing 
medium. If imtruments used in the evaluation of online dehvered courses have previously been validated 
only for use in the traditional classroom, confidence can be diminished in the feedback generated by those 
evaluations by online faculty. 

Method 

As World Wide Web courses became more widely implemented and include the participation of more 
feculty members at West Texas A&M University, concerns were raised by the faculty teaching online 
courses as to whether the ev^uation of courses delivered via the World Wide Web would be comparable to 
the evaluation of courses delivered in the more traditional manner. Due to the dearth of World Wide Web 
course delivery evaluation instrumentation in the hterature, the need for a delivery method specific course 
and mstructor evaluation instrument became clear to both those who administrate and those who taught 
online courses at West Texas A&M University. The first goal in developing such an instrument was to 
provide for a comprehensive validation strategy through which the validity of the instrument could be 
quantified procedurally and later statistically. 

This study was pursued in two main phases. In the first phase, a panel of web courseware development 
faculty, administrators, and staff were gathered at West Texas A&M University in the M semester of 1998 
for the purpose of developing a course evaluation instrument. The expert panel assembled consisted of 
LeAnn McKinzie, Associate Dean for Information Technology, James T^at, assistant professor of 
instructional technology and research, Charles Mize, program head of instructional technology, and Trey 
McC^e, instru^onal tectoology graduate student and WTOnline university network programmer. This 
panel met to d^ide on the initial form of the instrument and questioning domains to be used in gathering 
data Th^ main questioning domains were chosen; they included questions designed to elicit perceptions 
as to delivery technology , course content, and instructor. The questions in the delivery technology domain 
sought infonmtion from students ^ to how they perceived the role of the Internet-based technologies used 
to ^liver online courseware in their overall course experience. The course content domain questions were 
written to wUect ^ta on students’ perception of the quality, structure and value of the course content as 
they experienced it through the online course presentation. Lastly, the instructor evaluation domain sought 
to determine how students viewed the contributions, ejq>ertise, and responsiveness of the instructor. 
Incorporate into the fii^ draft were instructor and course content questions that were reflective of the five 
main questioning domains of the CiiiQ (Aleamoni, 1978) instrument used to evaluate classroom-based 
courses (course value, method of instruction, instructor attributes, course material, and student interest) 

The se<^d phase of the study began with the completion of the first draft of the online courseware 
evaluation instrument This instrument was distributed among several instructors of courses that were 
offered throi^ World Wide Web delivery for comment. These comments were evaluated and, when 
appropriate, incorporated into the instrument The draft instrument was then distributed to a class of 
graduate students at West Texas A&M University. The students participating in this pilot study were asked 
to complete the survey instrument and check for proper wording, phrasing, and time of completion. 



This second draft of toe instrument was then rewritten as a World Wide Web based point and click form 
and posted on toe university server which hosts online courseware. Instructors of online courses were 
requested to encourage their students to participate in toe study by evaluating toe courses in which they 
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were currently enrolled using this instrument Additionally, all students currently enrolled in World Wide 
Web based courses were notified by email of the existence of the instrument md were asked to use &e 
instrument to evaluate the course or courses they were currently taking. A point and click World Wide Web 
address was included in the email sent to all students enrolled in online courses to enable the students to 
easy Icxjate the web site where the instrument was posted. 

Data gathered from the spring 1999 semester field study was used to refine and further focus the instrument 
used for distribution to summer online courses. Similarly data gathered fix)m two summer session 
distributions was used to further refine the instrument distributed in the fall semester of 1 999. Data 
collected in each of the three field trials of the instrument have been analyzed to characterize the 
perceptions of study participants of their online course experiences. Also gathered data was used to 
discover the strengths and weaknesses of the online evaluation instrument, and the use of the Internet to 
evaluate student course perceptions. At the time of the writing of this paper data has been gathered from the 
fourth distribution of the instrument. The accumulated data will be us^ to perform a factor analysis of the 
questions used to quantify the validity of the instrument 

Findings 

The process through which the instrument was developed progressed through two expert panels (research 
and development and online teaching faculty), and both a small and a large-scale pilot study. When the 
data from the first three distributions (Spring, Summer I, and Summer II semesters of 1999) of the 
instrument were analyzed to determine a reliability coefficient for the questions used to gather data fix)m 
the three questioning domains, a standardized item alpha coefficient of .7784 was obtained 

Data collected through the summer semester included 204 usable responses reflecting a total response rate 
of ^jproximately 30% of students who were requested to participate in the study. At the writing of this 
article approximately 185 further responses have been gathered fix>m students enrolled in ^ semester 
1999 courses, a response rate of approximately 35%. Of those students responding, approximately 28®/o 
were male, and 72% were female. Students responding to the survey were from a range of ages, including 
38% from 23-30 years of age, 27% from 31-40 years of age, and 18% from 41-50 years of age. The 
remaining 17% of respondents were either unda* the age of 22 (14%) or over the age of 50 (3%). Tbs 
result is affected by the fact that West Texas A&M University has a strong population of iion-traditional 
students, and that graduate courses were predominant (graduate 64%, undergraduate 34% in respondents) 
in the online offerings of the university in the early stages of its developmrat. The majority (53%) of 
students responding to the survey instrument reported having taken no online courses previous to the course 
they were evaluating, while an additional 3 1% of respondents reported having taken either one or two 
classes previously. Respondents to the study reported confidence in their mastery of course material taught 
as is evidenced by the fact that 71% reported that they expected a grade of “A^ while 27% expected a “B.” 
Only 2.5% reputed an expectation of a grade less than “B.” When respond^ts were asked how long they 
spent convicting the survey, 96% of respondents reported that it took 15 minutes or less, while 74% 
reported taking 10 minutes or less. 

Although text boxes were provided for respondents to comment v^on the instrument itself, there were no 
comments or suggestions submitted. Similarly there were no comments added by respondents concemmg 
questions that should be added or deleted Respondents did, however, use the text boxes provided to 
comment on their instructor and/or their course. 



Condnsions 

Many lessons were learned by those researchers who woiked together on this project di^g the 1998-1999 
study period Chief among these lessons is the finding that the traditional method of inviting particip^on 
in a study that is facilitated through technological means (i.e. the submission of a world wide web point and 
click form after notification by email) has not resulted in the rate of return desired by the researchers. WWe 
this low rate of return is not as significant an issue during the pilot testing phase of a validation study as it 
would be \dien the instrument, the researchers has hoped for a higher rate of return overall. In the 
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traditional classroom setting achieving a high rate of return on such an evaluative instrument is relatively 
straightforward, collecting such data through the internet has the potential to be somewhat more 
challenging. The maximization of student participation in online course evaluation will depend iqx)n the 
implement^on of new technologically enhanced methods developed and used to compel students to 
participate in an online-based course evaluation regardless of type. The use of such methods will allow the 
return rate of online course evaluations to ^proach or even exceed the return rate of evaluative instruments 
administered in the classroom. 

A second lesson leam^ was that in any process which has as its goal the development of an instrument for 
the purpose of evaluating instruction there is a need for involvement of all effected groups in the 
development process. Those researchers who wish to develop such an online course specific evaluative 
instrument would be well served to involve the fk:ulty, both those teaching on-line and those not, in the 
development process. Those faculty members who are teaching online will most probably be doing so 
without great experience in using Internet and World Wide Web technologies and media, and wiU view 
some trepidation evaluative methods that will affect their overall teaching effectiveness ratings. 
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